fix(index): every xsd:double value is indexable by aaj3f · Pull Request #2044 · fluree/db

aaj3f · 2026-10-08T23:49:51Z

Summary

Fixes #2043

xsd:double and xsd:float include the special values INF, -INF and NaN (XSD 1.1 Part 2 §3.3.4, §3.3.5). With this change every double has an index key, the special values read back on every lane with SPARQL's value semantics (export included), and xsd:double/xsd:float lexical forms are parsed as XSD defines them.

What changes

Index keys. ObjKey::encode_f64 gives every double a key. The keys extend the existing order inside NUM_F64: -INF below every finite value, INF above, and one canonical NaN above INF. The index format and its version are unchanged, and existing index files need no migration.
One storage order. FlakeValue's Ord and the index keys share that total order, so novelty and the index sort doubles alike. The overlay, scan and bound-lookup paths encode the special values, and bulk import stores them as numbers, as transactions do.
Order and comparison are separate. Storage order (index keys, FlakeValue::cmp, ORDER BY) is total, with NaN after INF and all NaNs equal. SPARQL comparison follows XPath, under which NaN is unordered. Numeric range walks (the COUNT, top-k and range-semijoin fast paths) stop at INF's key, so no comparison selects a NaN.
Lexical forms. One parser (fluree_graph_ir::parse_xsd_double / parse_xsd_float) defines the lexical space: numerals, INF, +INF, -INF and NaN, with numerals beyond the range mapping to INF/-INF as XSD 1.1 specifies. Every write surface, query literals, casts and string-backed aggregate inputs use it. Rust's spellings (inf, Infinity, nan) are not values: JSON-LD and SPARQL UPDATE refuse them as ill-typed literals and Turtle keeps them as ill-typed literals, the rule other built-in datatypes follow (fix: keep typed literal values through the index (reindex affected ledgers) #1988). SPARQL UPDATE words that refusal as JSON-LD does (Parse error: Cannot parse 'inf' as xsd:double: …). R2RML renders non-finite float cells with the XSD spellings, and the fused R2RML aggregate reads a double text cell with the same parser (other text sends the query to the generic pipeline). Cross-ledger shapes carry INF/-INF to the data ledger in XSD form. A bad xsd:double query constant is refused as Invalid xsd:double literal 'inf': …. Cypher's toFloat is not an XSD cast and keeps reading strings as Cypher does, Infinity, -Infinity and NaN included.
Indexer. A failure to resolve a commit's content into index records is now IndexerError::Unindexable. The same build would stop the same way, so the background indexer halts the ledger: waiters resolve as failed, index status reports halted: true with the error, and an incremental build does not fall back to a full rebuild for it. Implicit triggers (a commit, a push, a published commit, a catch-up sweep) start no build for a halted ledger. An explicit index request (IndexerHandle::trigger_explicit, used by the admin index API), a reindex that succeeds, or a restart lifts the halt; so does a newer index that another process publishes for the ledger, once the catch-up re-sweep sees it (a worker that runs no catch-up sweeps does not). Dropping a ledger through this process clears its halt, so a ledger created again under the same id starts without one. The halt is logged at error, and a write refused because novelty is full on a halted ledger is logged at error with the halt's cause.

Semantics

Per the specs, except where marked:

=, <, >, <= and >= are false when an operand is NaN, and != is true (SPARQL 1.1 §17.3; XPath F&O 3.1 §4.3). The comparison is false, not a type error, so !(?v < 0) holds for NaN.
INF is above and -INF below every finite double (F&O 3.1 §4.3.2).
Implementation-defined: Fluree compares numbers exactly, so an integer or decimal beyond the double range is still below INF. XPath would promote it to a double, making 10^400 = INF true.
A NaN literal in a triple pattern matches stored NaN values, since pattern matching is by term (§18.3.1), and sameTerm(NaN, NaN) is true (§17.4.1.8).
SUM and AVG with a NaN member are NaN, and INF + -INF is NaN (§18.5.1.3, §18.5.1.4).
COUNT counts NaN (§18.5.1.2). DISTINCT, GROUP BY, COUNT(DISTINCT) and hash joins treat every NaN as one value, whatever its sign or payload (§18.5, §18.5.1), and -0.0 as 0, on every read lane.
SHACL sh:hasValue and sh:in treat NaN as one term; no range facet holds for NaN.
Implementation-defined: ORDER BY places NaN after INF in ascending order; SPARQL leaves its position open (§15.1). MIN and MAX follow ORDER BY (§18.5.1.5, §18.5.1.6), so MAX is NaN when a NaN is present and MIN is the smallest other value.

docs/concepts/datatypes.md gains a section with the same list.

Compatibility

Persisted formats: unchanged. No commit, index, nameservice or content-identity format changes (root FIR6 v2, leaf FLI3 v1, branch FBR3 v1). Checked with the v4.2.3 release CLI against this branch:
- on an index v4.2.3 built (plus novelty), 19 queries return byte-identical results with this branch and with main;
- index builds and full rebuilds of the same pre-existing data produce byte-identical leaf, branch and dictionary files with this branch and with main;
- an incremental build on a v4.2.3 index adds files and rewrites none, with the same file headers.
A string object under xsd:double/xsd:float is read as a number only when it is an XSD lexical form. Such strings come from SPARQL UPDATE xsd:float literals written before fix: keep typed literal values through the index (reindex affected ledgers) #1988; a non-XSD spelling among them now reads as the ill-typed literal it was stored as.
Existing indexes read as before, including xsd:double/xsd:float values that a previous bulk import stored in the index as text (inf, -inf, NaN). Arithmetic, SUM, AVG, casts to xsd:double/xsd:float and a comparison inside a larger expression read that text as the value; to isNumeric, ORDER BY, MIN/MAX and a FILTER that is a single comparison it stays text, as before. One answer changes: SUM/AVG over NaN text are NaN, as SPARQL specifies, where they were unbound. Only reads take the inf/-inf spellings; the write surfaces keep to the XSD lexical space, though a literal "inf"^^xsd:double written today is read the same way, since it cannot be told apart. A full reindex stores the values as numbers. docs/concepts/datatypes.md describes the exception.
Rust API: ObjKey::encode_f64 is infallible, and ObjKeyError loses its NaN/Infinite variants. IndexerError gains Unindexable; IndexStatusSnapshot and IndexStatusResult gain halted; IndexerHandle gains trigger_explicit and clear_halt. The indexer gains a test-utils feature with halt_for_test and build_incremental_index_for_record (an incremental build with no fallback to a full rebuild). Function gains CypherToFloat. fluree_db_core::coerce gains XSD_DOUBLE_LEXICAL_HINT.

Performance

Measured against main at 6b9d5b619 (this branch as it stood on that base) on a numeric-heavy ledger: 1.5M triples (500k subjects, each with two xsd:double values and one xsd:integer), both CLIs built with the dev-fast profile, runs interleaved, medians of 5 runs (15 for the last three reads, 7 for the novelty insert):

	`main`	this branch	ratio
bulk import	3.26 s	3.25 s	1.00
full rebuild (`reindex`)	1.86 s	1.88 s	1.01
novelty insert, 50k subjects (7 runs)	0.503 s	0.468 s	0.93
range select, `1e5 < ?v < 1e10`	0.170 s	0.166 s	0.98
`COUNT` with `?v > 0` over index + novelty	0.236 s	0.238 s	1.01
`ORDER BY ?v` over 550k values, index + novelty	0.658 s	0.666 s	1.01
`SUM` / `AVG` over index + novelty	0.969 s	0.994 s	1.03

Every other measured read (fast-path COUNT, top-k, MIN/MAX, GROUP BY, bound lookup, SUM/AVG over the index alone) is within ±2.5% at its final run count; two 5-run cells that read +7% and +8% were re-measured at 15 runs and did not reproduce. Query results are byte-identical between the two builds.

Tests

it_xsd_double_special_values (grp_index): values written through JSON-LD, SPARQL UPDATE, Turtle and bulk import read back on every lane, including after retraction; one transaction holding every numeric type; the semantics table on SPARQL and its JSON-LD twin, on four read lanes (novelty, novelty over an index, incremental build with no fallback to a rebuild, full rebuild); ordering, comparison, COUNT(DISTINCT), MIN/MAX and DATATYPE among integers beyond i64; GROUP BY, COUNT(DISTINCT) and hash joins over NaNs of either sign and over -0.0, on both surfaces and every lane; values copied out of a named graph by INSERT … USING and retracted through DELETE … WHERE, including WITH a named graph; non-XSD spellings per surface, and the query-constant error; casts; export of the special values (N-Triples and JSON-LD) from novelty, from novelty over an index and from the index; an xsd:float numeral beyond the float range.
it_xsd_double_fast_paths (own binary): the COUNT-with-comparison, MIN/MAX and top-k fast paths with the kill switch on and off, routing pinned on both index lanes (COUNT takes its fast path over novelty too; MIN/MAX and top-k leave it to the general pipeline there); integer thresholds beyond 2^53 that round down and up.
it_index_unindexable (own binary): a ledger the v4.2.3 CLI wrote with two vector lengths on one property (tests/fixtures/vector-lengths-4.2.3), indexed by the background indexer, halts on the incremental path without falling back to a full rebuild, and halts on the full-rebuild path; a write refused once novelty is full logs the halt.
it_shapes_cross_ledger: a model ledger's shape with INF/-INF bounds and sh:in members is enforced on the data ledger.
it_query_cypher: toFloat reads Infinity, -Infinity and NaN.
Unit tests: lexical parser, coercion, key order, the comparison outcomes, aggregates, key intervals, walk bounds, resolver classification, the indexer's halt and its explicit retry, equal-value hashing of doubles (join keys, group keys, FlakeValue, the reasoner's derived facts), SHACL value constraints, R2RML rendering, the fused aggregate and Cypher's toFloat.
values_a_previous_bulk_import_stored_as_text_read_as_before (grp_index): reads an index that v4.2.3's bulk import wrote (tests/fixtures/bulk-import-4.2.3, with its source Turtle) on both surfaces and expects v4.2.3's own answers (but for SUM over NaN text), including NaN text, casts, and filters inside an expression, over a UNION and an OPTIONAL; a reindex of that fixture stores the values as numbers. The fixture's README records the commands that made it. inf_text_reads_as_infinity_in_arithmetic_and_aggregates: the same text written as Turtle, from novelty and once indexed.
it_index_halt (grp_index): a reindex that succeeds lifts a halt; a ledger dropped while halted and created again under the same id indexes its next commit. Orchestrator unit tests: implicit triggers leave a halted ledger alone, an explicit trigger builds it, a newer index lifts the halt, and cancelling a ledger clears its halt while a maintenance hold keeps it.
Unit tests: equal numbers hash equally across Long, Double, BigInt and Decimal; Long(i64::MIN) compares with doubles; graph-ir literal order is total and agrees with equality.

Commits

feat(graph-ir): one parser for the xsd:double/xsd:float lexical space, used by every surface.
fix(index): every double has an index key; one total storage order.
fix(query): SPARQL comparison, sameTerm and aggregates for the special values; group and join keys by value.
feat(indexer): a ledger whose committed data cannot be indexed is reported as halted; implicit triggers leave it halted.
test(api): integration tests.
docs(datatypes): the special values and their query semantics.
fix(query): INF and -INF that an earlier bulk import stored as text read as before.
feat(indexer): a newer index lifts a halt.
fix(core): Long(i64::MIN) compares with a double by magnitude.
fix(core): Double(2^63) hashes like the integer it equals.
fix(graph-ir): double literals order by IEEE total order.
fix(indexer): dropping a ledger clears its halt.
fix(api): cross-ledger shapes carry INF and -INF in XSD form.
fix(sparql): a bad xsd:double query literal names its datatype.
fix(query): fast paths compare an integer threshold with doubles exactly.
fix(query): the fused R2RML aggregate reads double text cells as XSD.
test(indexer): test hooks behind a test-utils feature; pin the incremental lane.
test(api): unindexable committed data halts both build paths.
fix(indexer): a halt is logged as an error, and named where writes are refused.
fix(query): casts read imported INF text as before; scope the datatypes docs.
test(api): export writes the special values from novelty and the index.
docs(query): correct a comment on NaN in the overlay walk.

Follow-ups

Follow-up: Values stored with less detail than written: re-upserts, sub-microsecond temporals, and one identity in novelty and the index #2090 — xsd:float values are stored as doubles without single-precision rounding, so "3.4e39"^^xsd:float keeps its double value while the cast xsd:float("3.4e39") gives INF. Rounding every float literal changes stored values and equality, so it is left for a separate change.
Follow-up: Special double values on the remaining surfaces: rendering, division, casts, JSON-LD filter atoms, coercion, MEDIAN/VARIANCE/STDDEV #2073 — MEDIAN, VARIANCE and STDDEV leave NaN out, while SUM and AVG propagate it (making them consistent also needs MEDIAN's sort on the total order), and the special values on the remaining surfaces: null in commit details and SHACL sh:value, double division by zero, xsd:boolean(NaN), JSON-LD filter s-expression atoms, and the double-to-integer bound in coercion.

…pace `parse_xsd_double` and `parse_xsd_float` accept exactly the XSD 1.1 lexical forms: numerals, `INF`, `+INF`, `-INF` and `NaN`. A numeral too large in magnitude for the datatype maps to `INF` or `-INF`, as XSD 1.1 specifies. Rust's spellings (`inf`, `Infinity`, `nan`) are not lexical forms of either datatype. Every place that turns a double or float literal into a value uses it: JSON-LD coercion, SPARQL query literals and casts, SPARQL UPDATE, Turtle, and string-backed aggregate inputs. JSON-LD and SPARQL UPDATE refuse the other spellings as ill-typed literals, the rule other built-in datatypes follow (#1988), and Turtle keeps them as ill-typed literals. SPARQL UPDATE words the refusal as JSON-LD does (`Parse error: ...`). R2RML renders non-finite float cells with the XSD spellings. Cypher's `toFloat` is not an XSD cast. It gets its own function and reads strings as Cypher does, `Infinity`, `-Infinity` and `NaN` included.

`ObjKey::encode_f64` is total. `-INF` sorts below every finite value, `INF` above, and NaN, canonicalized to one quiet NaN, above `INF`. The keys extend the existing order inside `NUM_F64`, so the index format and its version are unchanged and existing index files need no migration. `FlakeValue`'s `Ord` and the index keys share this one total order, so novelty and the index sort doubles alike. The overlay, scan and bound lookup paths encode the special values, and bulk import stores them as numbers, as transactions do. Numeric range walks (the COUNT, top-k and range-semijoin fast paths) stop at `INF`'s key, since no comparison selects NaN. `ObjKeyError` loses its `NaN` and `Infinite` variants.

Storage order is total, but SPARQL comparison is XPath's (SPARQL 1.1 §17.3, F&O 3.1 §4.3): `=`, `<`, `>`, `<=` and `>=` are false when an operand is NaN, and `!=` is true. The comparison is false, not a type error. `INF` is above and `-INF` below every other number. `sameTerm(NaN, NaN)` is true (§17.4.1.8). `SUM` and `AVG` with a NaN member are NaN, and `INF + -INF` is NaN (§18.5.1.3, §18.5.1.4), on the generic and the fused R2RML aggregate lanes alike. SHACL `sh:hasValue` and `sh:in` treat NaN as one term, and no range facet holds for NaN. GROUP BY, COUNT(DISTINCT), hash joins and the reasoner's derived-fact set key a double by its value, as `FlakeValue`'s equality and hash do: every NaN is one key, whatever its sign or payload, and -0.0 is 0.0.

A failure to resolve a commit's content into index records is now `IndexerError::Unindexable`. The same build would stop the same way, so the background indexer halts the ledger: waiters resolve as failed, index status reports `halted: true` with the error, and an incremental build does not fall back to a full rebuild for it. An implicit trigger (a commit, a push, a published commit, a catch-up sweep) starts no build for a halted ledger; its waiter resolves as failed with the same error. An explicit request (`IndexerHandle::trigger_explicit`, used by the admin index API), a reindex that succeeds, or a restart clears the halt. `IndexStatusSnapshot` and `IndexStatusResult` gain `halted`.

…lane - `it_xsd_double_special_values` (grp_index): values written through JSON-LD, SPARQL UPDATE, Turtle and bulk import read back from novelty, novelty over an index, an incremental build and a full rebuild, including after retraction; the SPARQL value semantics on both query surfaces and every lane; GROUP BY, COUNT(DISTINCT) and hash joins over NaNs of either sign and over -0.0; ordering and DISTINCT among integers beyond i64; one transaction holding every numeric type; copies through INSERT ... USING a named graph and retraction through DELETE ... WHERE (with WITH); non-XSD spellings per surface. - `it_xsd_double_fast_paths` (own binary): the COUNT-with-comparison, MIN/MAX and top-k fast paths agree with the general pipeline, with routing pinned. - `it_query_cypher`: `toFloat` reads `Infinity`, `-Infinity` and `NaN`.

Bulk import in earlier versions stored xsd:double and xsd:float INF and -INF in the index as the text `inf` and `-inf`. Arithmetic, SUM, AVG and comparisons inside expressions read that text, under those two datatypes, as INF and -INF, as they did before, so such an index reads as it did without a rebuild. Only reads take these two spellings; the write surfaces keep to the XSD lexical space. Tests: an index that v4.2.3's bulk import wrote (fluree-db-api/tests/fixtures/bulk-import-4.2.3, with its source Turtle), read on both query surfaces against v4.2.3's own answers; the same text written as Turtle, read from novelty and once indexed; unit tests at both read sites. docs/concepts/datatypes.md describes the exception.

A halted ledger records the nameservice `index_t` its failed build started from. The periodic re-sweep lifts the halt when the ledger's `index_t` has moved past it, as it does when another process indexes the ledger; implicit triggers then build it again. Tests: the re-sweep lifts a halt once a newer index is published, and not before; a reindex that succeeds lifts a halt (through a hidden test hook, `IndexerHandle::halt_for_test`).

`i64_fits_f64` compares the unsigned magnitude, so `Long(i64::MIN)` compares with any double by value, like every other integer beyond 2^53.

`FlakeValue` equality is by value across numeric types, so `Double(2^63)` equals `BigInt(2^63)`. Its hash took the integer path for doubles up to `i64::MAX as f64`, which rounds up to 2^63, and so hashed 2^63 as `i64::MAX`. The integer path now stops below 2^63, and 2^63 hashes as the exact decimal the `BigInt` arm uses. The test checks that equal values hash equally across `Long`, `Double`, `BigInt` and `Decimal`.

`LiteralValue`'s `Ord` compares doubles with `f64::total_cmp`. That order is total over every double, NaN of either sign included, and it is `Equal` exactly when the bits are, as `PartialEq` compares them, so sorting a graph's triples is well defined.

`IndexerHandle::cancel`, which every drop path calls, now also clears a halt and its error: it ends the ledger's indexing state, so a ledger created again under the same id starts without one and its commits index as usual. Maintenance holds (a reindex, an index sweep) cancel pending work but keep the halt, since the ledger goes on. The test halts a ledger, drops it, creates it again under the same id, commits, and expects the index to advance.

bplatz

Approving — the core fix is solid: finite keys are unchanged (no migration), NaN/-0.0 canonicalization is consistent across key, Ord, Hash, group/join keys, range walks stop at INF, and the NaN comparison split never leaks the storage order into SPARQL comparison. Please review the inline comments before merging; a few items couldn't be anchored to the diff, so they're here:

Must-fix (not in the diff)

Cross-ledger shapes with INF/-INF now fail to translate — regression. fluree-db-api/src/cross_ledger/shapes_materializer.rs:297 renders FlakeValue::Double(f) => f.to_string(), which gives Rust's inf/-inf. The receiving side goes through value_convert::parse_xsd_lexical, which this PR switched to the strict parse_xsd_double, so a model ledger shape with sh:maxInclusive "INF"^^xsd:double (or -INF in sh:in) now errors with invalid …#double lexical \inf`. It round-tripped before. One-line fix: fluree_graph_ir::canonical_xsd_double(*f)`.
Issue linking. The body should say Fixes #2043, and the two Follow-ups (xsd:float single-precision rounding; MEDIAN/VARIANCE/STDDEV vs NaN) need issues filed and Follow-up: #N lines (see CLAUDE.md / docs/contributing/issue-linking.md).

Should-fix

#2043 asks for export coverage ("Export drops the triple"). The fix for that comes from removing the NULL sentinel in dict_overlay/binary_scan, but no test exports NaN/INF from novelty and from the index.
The indexer halt (commits 4, 8, 12) is independent of the double fix — #2043 itself calls retry-forever "a separate problem". I'd be fine splitting it into its own PR; if it stays, see the inline comments on its test coverage and visibility.

Pre-existing, worth follow-up issues (not blockers)

graph_commit_builder.rs:112 and validate.rs:769/778 write null for INF/NaN (commit detail flakes; SHACL sh:value).
Double division by zero errors; op:numeric-divide says INF/-INF/NaN (and op:numeric-mod NaN).
xsd:boolean(NaN) is unbound; XPath casting gives false.
JSON-LD filter s-expression atoms still use parse::<f64>() (parse/filter_sexpr.rs:230,465), so (> ?v inf) / (= ?v nan) are doubles — contradicts "one parser on every surface".
coerce.rs:233: *d <= i64::MAX as f64 admits 2^63 and saturates to i64::MAX (the same off-by-one commit 10 fixed in Hash).
Stale comment at binary_scan.rs:~1705 saying NaN falls back to discriminant order in Ord.

bplatz · 2026-10-09T00:37:12Z

+    let (key, max) = match (otype, threshold) {
+        (OType::XSD_INTEGER, FlakeValue::Long(n)) => (ObjKey::encode_i64(*n).as_u64(), u64::MAX),
+        (OType::XSD_DOUBLE, FlakeValue::Long(n)) => (
+            ObjKey::encode_f64(*n as f64).as_u64(),


*n as f64 rounds a Long threshold above 2^53, so this lane doesn't compare exactly the way the generic lane does (numeric_cmp goes through BigDecimal). Reproduced: stored "9007199254740992"^^xsd:double, SELECT (COUNT(?s) AS ?c) WHERE { ?s ex:v ?v FILTER(?v >= 9007199254740993) } → 1 with fast paths on, 0 with them off.

Pre-existing, but this function was rewritten here and the PR body says "Fluree compares numbers exactly". Fix: compute the boundary key from the exact comparison (e.g. nudge to the adjacent f64 when n as f64 rounds across n), or decline the fast path when n.unsigned_abs() > 2^53. fast_star_const_order_topk.rs:342 has the same *n as f64.

Good catch, and it was worse than I'd have guessed. Rather than declining the fast path above 2^53, it now computes the exact boundary (5a474602f).

n as f64 gives the nearest double, and no double lies strictly between n and that neighbour. So the comparison only has to move to the neighbour's side of n: >= 2^53+1 becomes > 2^53, and > 2^53+3 becomes >= 2^53+4.

The top-k path's double filter now reuses that same key interval instead of its own *n as f64, so there's one place for this.

Tests:

A unit test sweeps thresholds around and past 2^53 (and i64::MIN/MAX) against an exact comparison, for all four operators.

it_xsd_double_fast_paths has your repro plus one that rounds up (stored 2^53 and 2^53+4, thresholds 2^53+1 and 2^53+3), on both surfaces, both index lanes, fast paths on and off.

Taking the adjustment back out fails both.

bplatz · 2026-10-09T00:37:12Z

+/// Add one row's floating value to the accumulator. NaN and the infinities are
+/// summed and counted like any other double, as the standard aggregate pipeline
+/// does: SUM is `op:numeric-add` (SPARQL 1.1 §18.5.1.3), so a NaN member makes
+/// the sum NaN. A physical exact or text column under a double datatype


"converts, as the generic path does" — the Column::String arm below (line 834) still uses t.trim().parse::<f64>(), which accepts Infinity, nan, +inf and whitespace. The generic lane now coerces strictly (coerce_value → parse_xsd_double), so a text cell Infinity gives SUM = INF here and unbound on the generic lane — a new disagreement between the two lanes (both accepted it before). Suggest using the same parser (plus legacy_import_infinity if you want parity with the index lane), without the trim. Separately (pre-existing), an unparseable cell is silently skipped here, whereas accumulate_exact_row escalates.

Agreed, and fixed in fab53f3be. A text cell under a double column now goes through parse_xsd_double plus legacy_import_infinity, with no trim. That's the same reading the generic lane gives an xsd:double string.

For anything else (Infinity, nan, a padded numeral) I didn't want to invent a third behaviour. The fold now reports the row, and the whole query re-runs on the generic pipeline: the same hand-off accumulate_exact_row already does on overflow, so the generic lane decides it. That also covers the silent skip you mentioned. The unit test covers the cells it reads and the ones it escalates.

bplatz · 2026-10-09T00:37:12Z

                    return Ok(result);
                }
+                // A full rebuild resolves the same commit and fails the same way.
+                Err(e) if !e.is_retryable() => return Err(e),


Nothing exercises a real Unindexable end to end. The orchestrator tests inject the error into on_build_error, and it_index_halt uses halt_for_test. If this arm, or the Resolve(Unindexable) → IndexerError::Unindexable mappings in build/incremental.rs (~730) and build/rebuild.rs (~365), regressed to a retryable error, every test would still pass and the ledger would go back to full-rebuilding forever. Could you add one test that commits genuinely unindexable data and asserts halted on both the incremental and the rebuild path, then mutation-prove it? Candidates: a vector dimension mismatch under one predicate (I couldn't find a write-side dims check — unverified), or a temporal/geo string that won't encode.

Added it_index_unindexable as its own binary (3cd0c95cc). Your hunch about the vector dimension mismatch was right: there was no write-side dims check. That's being fixed in its own PR, so the data here comes from a released version. tests/fixtures/vector-lengths-4.2.3 is a ledger the v4.2.3 CLI wrote; the README has the exact commands. It's indexed at t=1, with a vector of a different length on the same property at t=2.

With the real background indexer, the test asserts for both the incremental path and incremental-off that:

the build fails;

the ledger is halted with the error;

the incremental path did not fall back to a full rebuild (span capture).

I mutation-checked each of your three spots:

the non-retryable arm here → the fallback shows up and the test fails;

the Resolve(Unindexable) mapping in build/incremental.rs → same;

the mapping in build/rebuild.rs → the build retries instead of halting. The wait is bounded at 60s, so that fails rather than hanging CI.

bplatz · 2026-10-09T00:37:12Z

            return;
        }
+        if !error.is_retryable() {
+            warn!(


A halt is easy to miss in operation. This one warn! is the only signal; later implicit triggers log at debug!, and halted only reaches the Rust IndexStatusResult — the server has no index-status route, the CLI doesn't show it, and Python's index_status exposes error but not halted. What an operator sees is novelty growing until reindex_max_bytes, then 503 NoveltyAtMax on every write. Suggest error! here, and ideally naming the halt in the backpressure error (or exposing it on a status endpoint). Server operators' only levers today are /reindex (which fails again on genuinely bad data) or a restart.

Yeah, it was far too quiet. In 568ab4c40:

the halt itself logs at error!;

the novelty-full refusal path (request_index_after_novelty_rejection) now asks the indexer first. If the ledger is halted, it logs an error! with the halt's cause right where the 503s come from, and stops asking a halted ledger for a build it won't start. The end-to-end test checks for that line.

I didn't change what the client gets back, though. Callers (solo's transact classifier among them) match on NoveltyAtMax, and a distinct "halted" error needs its own status/retry decision. I'd rather make that decision on purpose than slip it in here. The same goes for a status surface: an index-status route, the CLI, and halted in Python's index_status. I'll get both written up as follow-ups.

bplatz · 2026-10-09T00:37:12Z

+    /// implicit trigger (a commit, a push, a published commit, a catch-up
+    /// sweep) starts one. An explicit request
+    /// ([`IndexerHandle::trigger_explicit`]), a reindex that succeeds, a newer
+    /// index that any process publishes for the ledger, dropping the ledger


"a newer index that any process publishes for the ledger … clears it" isn't unconditional: clear_superseded_halts only runs inside the catch-up re-sweep (line 2743), which is skipped when catchup_sweeps_enabled = false (the Raft delegated worker, fluree-db-server/src/state.rs:409-425) or catchup_interval is zero. Either call it on those paths too or qualify the docs here, in admin.rs:298-303 and in the PR body.

Related edge case (suspected): a drop by another process doesn't clear this process's halt, since only a local cancel does; a ledger recreated under the same id elsewhere stays halted here until its index_t passes the old one.

You're right that it wasn't unconditional. I went with qualifying the docs rather than adding a second lifting path (568ab4c40). IndexStatusSnapshot::halted, IndexStatusResult::halted and the body now say a newer index from another process lifts a halt only where catch-up sweeps run. A worker whose sweeps another worker in the process owns doesn't run them. An explicit index request, a successful reindex or a restart still lifts it everywhere.

On the drop-by-another-process edge: from reading the code (I didn't execute it), I think you're right that only a local cancel clears it. Elsewhere it waits for the re-created ledger's index to pass the halted one, or for a restart. The docs say that too now.

bplatz · 2026-10-09T00:37:13Z

-                    .map_err(|_| LowerError::invalid_decimal(value, datatype.span))?;
-                Ok(FlakeValue::Double(d))
+                fluree_db_core::coerce::coerce_string_value(value, dt_iri.as_str())
+                    .map_err(|_| LowerError::invalid_decimal(value, datatype.span))


Nit: a bad xsd:double/xsd:float literal now reports Invalid decimal literal 'inf'. Transactions say Cannot parse 'inf' as xsd:double. LowerError::invalid_literal(value, "xsd:double", …) would be accurate.

It now reads Invalid xsd:double literal 'inf': expected a numeral such as 1.5 or 1.5E3, or INF, +INF, -INF or NaN (77402823a). The hint is a single constant shared with the coercion error, so the two can't drift. The spelling test pins the message.

bplatz · 2026-10-09T00:37:13Z

@@ -12,6 +12,10 @@
 //! through these helpers so Fluree emits one consistent, spec-aligned form.
 //! JSON-LD (and other JSON-native typed output) is deliberately excluded:
 //! there a double is a native JSON number, never a lexical string.


Nit: stale — "there a double is a native JSON number, never a lexical string" contradicts the new docs: JSON-LD results carry the special values as the strings "INF"/"-INF"/"NaN".

Fixed (77402823a). It now says JSON-LD writes finite doubles as numbers and the special values as the strings "INF", "-INF", "NaN".

bplatz · 2026-10-09T00:37:13Z

+    fully_indexed(fluree, ledger_id).await
+}
+
+async fn incrementally_indexed(fluree: &Fluree, ledger_id: &str) -> LedgerState {


The "incremental build" lane isn't pinned to the incremental path. build_and_publish_index goes through build_index_for_record, which falls back to a full rebuild on any retryable incremental failure (lib.rs:242-247), so this lane could silently become a second full-rebuild lane and still pass. Could you assert that incremental actually ran (tracing span or result stats)?

Good call, that lane really could have quietly become a second rebuild lane. It's pinned now (9db9be46b): it builds through build_incremental_index_for_record (under test-utils), which runs the incremental builder with no fallback, so an incremental failure fails the lane.

bplatz · 2026-10-09T00:37:13Z

+                    }
+                }
+                let fired = proceeded.iter().filter(|s| *s == case.site).count();
+                if !disabled && lane == "indexed" && fired != 2 {


Nit: routing is pinned on the indexed lane and with fast paths off, but not on "novelty over an index". If the overlay lane is expected to take (or decline) the fast path, pin that too, so it can't pass by silently taking the generic lane.

Pinned (5a474602f). Each case now says whether its fast path also serves the novelty-over-an-index lane, and the test checks that. COUNT takes its fast path there; MIN/MAX and top-k hand it to the general pipeline. Answers are checked on both lanes either way.

bplatz · 2026-10-09T00:37:13Z

+
+/// Bulk import in earlier versions stored `xsd:double` and `xsd:float` `INF`
+/// and `-INF` in the index as the text `inf` and `-inf`. The fixture is such
+/// an index, written by v4.2.3's bulk import from `source.ttl` beside it. It


Nit: please record the exact v4.2.3 command(s) that produced tests/fixtures/bulk-import-4.2.3, so the fixture can be regenerated or extended.

Added tests/fixtures/bulk-import-4.2.3/README.md with the two v4.2.3 CLI commands: fluree --direct init, then fluree --direct create ex:main --from source.ttl. The new vector fixture has a README the same way.

…ouble

The shapes materializer renders an `xsd:double` value with `canonical_xsd_double`, so a model ledger's `sh:maxInclusive "INF"`, or `-INF` in an `sh:in` list, reaches the data ledger as a lexical form the datatype defines and is enforced there. The Turtle-to-JSON adapter likewise spells a numeral beyond the double range `INF`/`-INF`.

A query constant such as `"inf"^^xsd:double` is refused with `Invalid xsd:double literal 'inf': expected a numeral such as 1.5 or 1.5E3, or INF, +INF, -INF or NaN`, the hint transactions give, instead of calling it a decimal. The hint is one constant, `XSD_DOUBLE_LEXICAL_HINT`. The `xsd_double` module comment now says how JSON-LD writes the special values.

An `i64` beyond 2^53 may have no double of its own, and `n as f64` rounds it. The COUNT-with-comparison fast path now compares with the double `n` rounds to and adjusts the comparison to that double's side of `n`; no double lies between them, so it selects exactly the rows the general pipeline does. The top-k fast path's double filter reuses the same key interval. `it_xsd_double_fast_paths` gains the cases (stored 2^53 and 2^53 + 4, against 2^53 + 1, which rounds down, and 2^53 + 3, which rounds up; both surfaces, both index lanes), and pins routing on the novelty-over-an-index lane as well: COUNT takes its fast path there, MIN/MAX and top-k leave it to the general pipeline.

A text cell under an `xsd:double` column is read with `parse_xsd_double` (and the `inf`/`-inf` text of earlier imports), as the generic lane reads an `xsd:double` string, with no trimming. Any other text (`Infinity`, `nan`, a padded numeral) is no longer read the Rust way or skipped: the fold reports it, and the query re-runs on the generic pipeline, which decides it there.

…ental lane `IndexerHandle::halt_for_test` (and `TriggerHandle`'s) now exist only with the indexer's `test-utils` feature, which `fluree-db-api` enables for its tests. The feature also adds `build_incremental_index_for_record`, an incremental build with no fallback to a full rebuild. `it_xsd_double_special_values` builds its incremental lane through it, so that lane fails rather than quietly running a full rebuild.

`it_index_unindexable` (own binary) indexes a ledger that the v4.2.3 CLI wrote, indexed at t=1 and holding at t=2 a vector of another length on the same property (`tests/fixtures/vector-lengths-4.2.3`, with the commands that made it). With the background indexer it asserts, on the incremental path and with incremental builds off, that the build fails, the ledger is halted with the error, and the incremental path does not fall back to a full rebuild (span capture).

…e refused The background indexer logs a halt at `error`, not `warn`. When a write is refused because novelty is at its maximum and the ledger's indexing is halted, the API logs that at `error` with the halt's cause, where the refusals happen; it no longer asks a halted ledger for a build it would not start. The docs of `IndexStatusSnapshot::halted` and `IndexStatusResult::halted` now say that a newer index from another process lifts a halt only where catch-up sweeps run, which a worker whose sweeps another worker owns does not, and that a drop by another process is seen the same way. `it_index_unindexable` checks the refusal's log line.

…es docs `xsd:double(?v)` and `xsd:float(?v)` read a numeric typed literal held as text as arithmetic reads it, so the `inf`/`-inf` text an earlier bulk import stored casts to `INF`/`-INF` again, as it did before. `docs/concepts/datatypes.md`: - a numeral beyond the range is `INF` for `xsd:double`; `xsd:float` is stored at double precision, so `"3.4e39"^^xsd:float` keeps its value while the cast to `xsd:float` gives `INF`; - the imported-text exception covers `NaN` text, casts, and comparisons inside a larger expression, and says which filters keep the text; `SUM` and `AVG` over `NaN` text are `NaN`, where earlier versions left them unbound; a literal `"inf"^^xsd:double` written today reads the same way; - comparing an integer or decimal beyond the double range with `INF` is a choice: Fluree compares exactly, where XPath would convert it to `INF`. Tests: the v4.2.3 fixture test adds NaN text in arithmetic and SUM, the casts, and filters inside an expression, over a UNION and an OPTIONAL (v4.2.3's own answers, but for the NaN SUM); a reindex of that fixture stores the values as numbers; the float-range behavior. The fixture's README records the commands that made it.

`export_writes_the_special_values`: N-Triples carries each value as `"INF"^^xsd:double` (and `-INF`, `NaN` twice, beside the finite ones), and JSON-LD carries the strings `"INF"`, `"-INF"` and `"NaN"`, from novelty, from novelty over an index, and from the index after a full rebuild.

`FlakeValue`'s `Ord` is a total order over numbers, NaN after `INF`, so the bounded overlay walk's comment no longer says NaN falls back to discriminant order.

aaj3f · 2026-10-12T16:14:42Z

Thanks for such a careful pass on this one. Everything is addressed in ten commits on top of your merge (f58616ffb..835ae88c4). I had Claude work through the list with me, and nothing earlier was rewritten.

On the must-fix items, the shapes materializer now renders doubles with canonical_xsd_double (f58616ffb). There's a test that a model ledger's shape with sh:maxInclusive "INF", sh:minExclusive "-INF" and -INF inside an sh:in list is actually enforced on the data ledger. Claude swept for other places that render a double with Rust's to_string() and hand it to the strict parser, and found one more: the Turtle-to-JSON adapter spelled an overflowing numeral like 1e400 as inf, so that's in the same commit. The body now says Fixes #2043, and the two follow-ups have Follow-up: lines (#2090 for xsd:float precision, #2073 for MEDIAN/VARIANCE/STDDEV and the other surfaces).

On the should-fix items:

Export: export_writes_the_special_values checks N-Triples and JSON-LD from novelty, from novelty over an index, and from the index (0de4879a0). Putting the NULL sentinel back in the scan path fails it.
The halt: I kept it in this PR. It felt simpler to land it with the test coverage you asked for than to split it today. I took all of your inline items on it, details in each thread.

The pre-existing items you listed are in #2073 as one checklist: the null writers, double division by zero, xsd:boolean(NaN), the filter s-expression atoms, and the coerce.rs 2^63 bound. The exception is the stale NaN comment in binary_scan.rs, which was one line, so I just fixed it here (835ae88c4). The Long-threshold rounding and the fused aggregate's text parse were close enough to this PR that I folded them in rather than filing them.

One conflict, in fluree-db-query/src/eval/compare.rs: #2044 made the ordering comparison return a ValueOrder (a NaN operand compares false, not as a type error), and this branch replaced the plain-string temporal coercion with temporal::cmp_with_lexical. Kept #2044's ValueOrder and double handling as they are; the coercion calls cmp_with_lexical and returns ValueOrder::Ordered.

Brings in #2019 and #2044. One conflict, in `eval/rdf.rs`, where both sides added next to `fast_same_term_encoded_ids`: #2044's `same_term` (two evaluated values are one term when equal, every NaN the one term "NaN"^^xsd:double) is kept as it is, and `eval_same_term` calls it; this branch's id comparison for encoded IRIs keeps its place ahead of both, and answers only when an encoded IRI is on one side, so doubles always reach `same_term`.

Brings in #2019, #2044, #2048, #2049 and #2052 (main at 58b5323). #2052 keys cross-ledger governance by the model ledger's head commit and opens the model at that commit. Two files conflicted: - tx.rs: this branch moved the stored-shapes pass into its own block (`stored_cache`), which #2052 edited in its old place. The block keeps this branch's shape, and its compiled-SHACL cache key carries #2052's commit id beside the model's t, as main's does. The shapes model opener and the cache key type merged without conflict. - it_policy_cross_ledger.rs: both sides appended tests; both are kept.

aaj3f added 12 commits October 8, 2026 07:40

docs(datatypes): xsd:double special values and how queries read them

a9ef7f8

fix(core): compare Long(i64::MIN) with a double by magnitude

783516e

`i64_fits_f64` compares the unsigned magnitude, so `Long(i64::MIN)` compares with any double by value, like every other integer beyond 2^53.

aaj3f added bug Something isn't working as expected area:storage Storage backends, nameservice, ledger lifecycle, index storage, caches labels Oct 8, 2026

aaj3f requested a review from bplatz October 8, 2026 23:49

bplatz approved these changes Oct 9, 2026

View reviewed changes

bplatz and others added 2 commits October 11, 2026 12:32

Merge remote-tracking branch 'origin/main' into fix/index-every-xsd-d…

b691c8d

…ouble

aaj3f mentioned this pull request Oct 12, 2026

Tracking: follow-ups deferred by #2006–#2010 and #2044 #2095

Open

aaj3f added 7 commits October 12, 2026 11:09

docs(query): correct a comment on NaN in the overlay walk

835ae88

`FlakeValue`'s `Ord` is a total order over numbers, NaN after `INF`, so the bounded overlay walk's comment no longer says NaN falls back to discriminant order.

aaj3f requested a review from bplatz October 12, 2026 16:15

This was referenced Oct 12, 2026

Show a halted index on every status surface (server, CLI, Python) #2099

Open

A write refused on a halted ledger should say the index is halted #2100

Open

release v4.2.4 #2041

Merged

aaj3f merged commit ba694ee into main Oct 13, 2026
19 checks passed

aaj3f deleted the fix/index-every-xsd-double branch October 13, 2026 17:07

aaj3f mentioned this pull request Oct 13, 2026

fix: JSON-LD graph scoping, config integrity and explicit-only SHACL #2009

Open

1 of 2 tasks

aaj3f mentioned this pull request Oct 13, 2026

fix(transact): write fidelity, part 2 — one fact identity, microsecond temporals, value reads #2109

Open

This was referenced Oct 13, 2026

fix(query): compare terms by what they are, not how they're encoded #2010

Merged

fix(transact): a property keeps one vector length per graph #2112

Open

Conversation

aaj3f commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What changes

Semantics

Compatibility

Performance

Tests

Commits

Follow-ups

Uh oh!

bplatz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aaj3f commented Oct 12, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

aaj3f commented Oct 8, 2026 •

edited

Loading